Below we discuss three key areas of related work. First, we discuss prior work that enables room-scale touch tracking. We then review room-scale approaches for tracking user location and pose. We conclude with systems able to detect and track objects. In particular, we focus primarily on systems that are deployed in the environment, as opposed to those that are carried (e.g., smartphones, wearables).
Room-Scale Touch
Most previous systems have achieved wall-scale touch sensing through optical approaches. For example, LaserWall used a scanned laser rangefinder operating parallel to a wall's surface to detect hand touches. Infrared emitter-detector arrays have also been used to create large interactive surfaces. Most popular are camera-based approaches, including invisible light, depth, and even thermal imaging.
People have also explored acoustic touch sensing approaches, for example, by attaching microphones to the corners of a desired interactive surface and using time difference of arrival methods. It is also possible to use an array of centrally located acoustic sensors for estimating the location of tap events. Researchers have also forgone absolute spatial tracking, and instead built interactions around gesture vocabularies.
More relevant to the systems disclosed herein are systems that use capacitive sensing. Early work by Smith et al. demonstrated a capacitive sensing wall able to detect user gestures such as swipes, though users had to stand on an active transmitter electrode. Living Wall offered discrete touch patches as part of an art installation. Electrick used electrical field tomography for coarse touch tracking, including a demo on a 4×8 foot sheet of drywall. To enable fine-grained finger interactions on furniture, researchers have used dense, self-capacitive electrode matrices.
Also, SmartSkin demonstrated a table-sized (80×90 cm) mutual capacitive matrix for touch sensing. We move beyond this seminal work with novel hardware and tracking algorithms, as well as a deeper exploration of electrode/antenna fabrication, especially as it relates to walls. We also uniquely consider interaction modalities at room scale.
User Tracking and Pose Estimation
There is extensive literature on indoor user localization. Technical approaches that instrument the environment include computer vision, floor pressure sensing, floor and/or furniture capacitive sensing, and RF sensing. Conversely, users can be instrumented with tags, such as RFID and Bluetooth beacons. There has also been substantial work on human pose estimation. Most common is to use cameras looking out onto an environment. Alternatively, cameras have been installed below the floor, as seen in GravitySpace and MultiToe, which used a room-sized FTIR floor to track users and infer posture. Beyond cameras, RF-based approaches are also popular, including Doppler radar, RFID tracking, and co-opting WiFi signals.
Most relevant to our systems are capacitive sensing methods. One of the earliest examples leveraging this phenomenon is the Theremin, a gesture-controlled electronic musical instrument. In HCI, researchers have frequently explored using capacitive sensing to detect the type and magnitude of body motion. For example, Mirage attached electrodes to a laptop to detect dynamic poses such as arm lifting, rotating and jumping. Valtonen et al. used two electrodes attached to the floor and ceiling to sense a user's height and thus can classify postures such as sitting and standing. Finally, Grosse-Puppendahl et al. explored posture estimation by instrumenting furniture with multiple electrodes, for example, a couch that can detect discrete postures such as sitting and lying.
Object & Appliance Sensing
Many systems have demonstrated appliance and tool detection using cameras. For example, Snap-To-It used a smartphone's camera to recognize and use appliances (e.g., an office printer). Maekawa et al. utilized wrist-worn cameras to detect what object was currently being used. Finally, Zensors leveraged crowd workers and machine learning to answer user-defined questions about environments, including appliances.
Another common approach is to sense sound or vibration emitted from operating appliances or objects. ViBand leveraged micro-vibrations propagating through a user's body for detection. Viridi Scope implemented a sensor tag featuring a microphone that can infer power consumption of an appliance. Similarly, UpStream attached a microphone to faucets for water consumption monitoring.
It is also possible to tag or mark an object for detection. For example, QR codes can be captured by cameras for object recognition. In addition, capacitive near-field communication has been used to augment objects with antennas for communication. Finally, RFID tags and Bluetooth beacons (as well as most work previously reviewed on user tracking), can also be adopted for object and appliance sensing.
Finally and closest to our sensing principle are approaches that take advantage of EM noise generated by appliances when active. This has been sensed previously by coupling to power lines and users' bodies, or by placing sensors proximate (≤10 cm) to appliances. As we will discuss, our method makes use of airborne EM signals, which enables appliance detection and tracking. We also significantly extend the sensing range beyond previous work, from centimeters to room-scale.
It is against this background that the techniques described herein have been developed.